Prior to setting forth the background of the invention, it may be helpful to set forth definitions of certain terms that will be used hereinafter.
The term ‘camera’ or ‘sensing device” as used herein is broadly defined as any combination of one or more sensors that are configured to capture three-dimensional data of a scene, directly or through further manipulation. An exemplary sensing device may include a pair of cameras which are configured to capture passive stereo which may derive depth data by comparing the images taken from different locations. Another example for a sensing device may include a structured light sensor which is configured to receive and analyze reflections of a predefined light pattern that has been projected onto the scene. A structured light system includes a radiation source (such as a laser source) and a sensor configured to capture the reflections from the radiation source Yet another example would be a single camera for capturing 2D images together with a measurement device that can measure the distance the sensing device travels. Such a measurement device can be, but not limited to, an inertial measurement unit (IMU) or odometer.
The term ‘three dimensional rig’ or ‘3D Rig’ as used herein is defined as any device for mounting at least two cameras or at least one camera and a radiation source that together form a 3D-system capable of capturing images and videos of a scene. A 3D Rig must provide the possibility to mount two cameras or one camera and one radiation source, with a horizontal or vertical offset and adjust the cameras in all possible axes.
The term ‘calibration’ and more specifically “camera calibration” as used herein is defined as the process of adjusting predefined parameters so that the camera, or the sensing device may operate optimally. Calibration of a 3D Rig involves the calibration of a plurality of cameras. The calibration of a 3D Rig includes the alignment or configuration of several cameras and in particular the process of identifying the deviation of several parameters relating cameras' spatial alignment from a predefined value tolerance and remedying the identified deviation. It is said that a certain sensing device is calibrated whenever the estimated calibration parameters, reflect the actual calibration parameters, taken at the specified timeslot, within an agreeable margin. It is possible to verify that a device is calibrated by comparing distances measured by the device with distances obtained from un-related sources (e.g., directly measured).
The term “landmark” as used herein relates to visual features used by a variety of computer vision applications such as image registration, camera calibration, and object recognition. Using landmarks is advantageous as it offers robustness with regard to lightning conditions as well as the ability to cope with large displacements in registration. As defined herein, a landmark comprises both artificial and natural landmarks. Exemplary landmarks may include corners or repetitive patterns in images.
One of the challenges of sensing devices that are required to capture a scene in real time, such as wearable near-eye displays, is to maintain calibration in real-time. More specifically, as such devices tend to lose its initial calibration quite easily (e.g., due to physical impact applied to the sensing device), it is essential to be able to regain calibration quickly without requiring manual intervention.
While calibrating a sensing device using pre-registered landmarks is well known in the art, when operating in an unfamiliar scene in which pre-registered landmarks cannot be identified, calibration becomes a more difficult task.
It would be, therefore, advantageous to provide a method and a system that addresses the calibration challenge in unfamiliar scenes.